Policy Evaluation

Change in government policy is brought about by influencing the opinions of leaders and decision-makers through policy advocacy. Policy advocacy can be defined as a wide range of activities conducted to influence decision-makers at various levels. It involves changing minds, reframing arguments, and inspiring social movements to influence public policy to deliver results at the community level. 

This kind of advocacy is achieved through various methods including litigation, lobbying, public education, capacity building, relationship building, forming networks, and leadership development. In low and middle-income countries (LMICs), programmes often focus on technical interventions and may not necessarily emphasize securing political commitment to the initiatives that have been undertaken. Advocacy and media outreach become crucial since awareness-building among the stakeholders and the beneficiaries is a major part of changing behaviors and catalyzing change.

Advocacy initiatives can be categorized into three forms:

1. Policy Advocacy: To engage individuals in positions of power
2. Systems Advocacy: To target established local bodies
3. Advocacy for behavior change among individuals or communities

To ascertain the effectiveness of policy advocacy initiatives, it is imperative to conduct policy evaluations. Policy evaluation can inform the implementing organizations and the donor about the process and results of advocacy initiatives and future initiatives. Changes through advocacy often occur in sudden leaps, in unexpected ways, and in response to the most unlikely circumstances. 

The efforts to evaluate advocacy and policy change are relatively recent compared to similar efforts in other fields. The policy is the unit of analysis for advocacy and policy change evaluations instead of the more usual programme or project. Identifying how changes in attitudes, behavior, or policy occur while acknowledging that this is not always a linear process is essential to comprehending the effect of advocacy campaigns. In general, advocacy evaluations gather data on three sorts of outcomes or results: advocacy ability, progress toward policy objectives, and the effect of advocacy activity.

In terms of methods, policy evaluation employs a mix of traditional and new-age evaluation concepts and methodologies to analyze the substance, execution, or effect of a policy. The evaluation can help in obtaining the value, worth, and usefulness of a policy. Policy evaluation, like any evaluation, may serve essential functions throughout the full chain of the policy process.

Policy Evaluation Objective:

It is important to mention that evaluating the development of a policy enables a better understanding of its background, substance, and execution. Its broader objective is to explore whether the policy is executed following its objectives and whether it achieves the desired results and impact. The specific policy evaluation objective can be further classified as:

• Does the content of the policy effectively communicate its objectives, execution, and underlying reasoning for why it would affect the desired change?
• Explore pathways of influence through an iterative theory of change: is there empirical support for key assumptions in the theory of change?
• Assessing the contribution to driving policy outcomes by developing a performance measurement system
• Determining what strategies have worked both in the interim and longer-term period: are there strategies across the different projects that have consistently worked?
• Explore the role of relationships in the influence pathways of the advocacy and the domain of opportunistic influence.

How a policy is implemented is essential to determine its efficacy. Evaluation of policy implementation may give critical information regarding the impediments and facilitators of implementation and a comparison of various implementation components.

Challenges in Policy Evaluations:

Policy evaluations are filled with challenges. The first challenge is that attributing or linking advocacy and identifying outcomes are complex. With advocacy, defining success could be tricky and may change depending on the timeline and nature of the project. The challenge is to convert subjective gains into objective reality. The advocacy evaluation can sometimes be unclear and poses challenges such as :

• Proving attribution is challenging, in part, due to the difficulty in constructing a robust counter-factual. Hence, it is difficult to prove that a policy change occurred primarily as a result of a specific organization’s work. It is useful to identify contributions like how an advocacy initiative helped to achieve/ influenced a policy change.  
• Decision-makers, who are the primary focus of advocacy efforts, may be unwilling to offer honest evaluations of the reasons that influence their policy decisions. This is why it is often difficult to get sincere feedback from policymakers about the efficacy of advocacy activity. In other circumstances, decision-makers may also be one’s competitors, which makes data collecting much more challenging.
• Context becomes key in the case of policy evaluation. It is important to understand the dynamics and factors that affect policy in its context. The frameworks shall explore how policy is formed and implemented through both formal and informal processes, and they explore the range of actors and interests to drive policy. The evaluation would attempt to understand how change occurs in each context, how strategies can be adapted to these contexts, and how results can be interpreted.

It is also critical to determine the role of the external environment and how it would be conducive to policy change. This includes changes in the political scenario, processes, or incentives, which can have a major impact on the demand for change.

References

Barry Coates & Rosalind David (2002) Learning for change: The art of assessing the impact of advocacy work, Development in Practice, 12:3-4, 530-541, DOI: 10.1080/0961450220149870

Coffman, J. (2009, October 13). A User’s Guide to Advocacy Evaluation Planning. Retrieved from http://www.pointk.org/resources/files/UserGuideAdvocacyEvaluationPlanning.pdf

Coffman, J. (2009). Overview of Current Advocacy Evaluation Practice. [online] Nyshealthfoundation.org. Available at: https://nyshealthfoundation.org/wp-content/uploads/2019/02/overview_current_eval_practice.pdf [Accessed 22 Aug. 2019].

Crewe, Emma & Harrison, Elizabeth (1998), Whose Development? An Ethnography of Aid, London: Zed. Crocker, David A

Dunn, William N. , 1994, Public Policy Analysis, Englewood Cliffs NJ: Prentice Hall

Fischer, Frank (1980), Politics, Values and Public Policy, Boulder, CO: Westview Press. —-, 1995. Evaluating Public Policy. Chicago: Nelson Hall.

Naeve K, Fischer-Mackey J, Puri J, Bhatia R, Yegbemey R N, (2017) Evaluating advocacy: an exploration of evidence and tools to understand what and why, Working Paper 29, International Initiative for Impact Evaluation .

Parish, S., Gladfelter, J., Reed, E., & Zandniapour, L. (n.d.). Measuring Influence: Advocacy Evaluation Challenges and Successes. Innovation Network, Inc.  Retrieved from http://www.pointk.org/resources/files/advocacy_eval_series_grantstation.pdf (Last accessed 22 August 2019)

Kultar Singh – Chief Executive Officer, Sambodhi